System and method for capturing digital images using multiple short exposures

ABSTRACT

Methods, devices, and computer program products for image sensors with overlapped exposure brackets supporting multiple short exposures are described. In one aspect, a method of capturing an image is disclosed. The method includes capturing, on a first subset of pixels on an image sensor, a first image with a first exposure length. The method further includes simultaneously capturing, on a second subset of pixels on an image sensor, a plurality of images with a second exposure length, wherein the second exposure length is shorter than the first exposure length. The method further includes combining the plurality of images with a second exposure length into a second image. Finally, the method includes combining the first image and the second image.

RELATED APPLICATION(S)

This application is a continuation of U.S. patent application Ser. No.14/152,888, filed Jan. 10, 2014, and entitled “SYSTEM AND METHOD FORCAPTURING DIGITAL IMAGES USING MULTIPLE SHORT EXPOSURES,” now U.S. Pat.No. 9,307,161, which is incorporated by reference herein in itsentirety.

FIELD

The present application relates generally to digital imaging, and morespecifically to systems, methods, and devices for image sensors withoverlapped exposure brackets supporting multiple short exposures.

BACKGROUND

In digital imaging, the dynamic range of a complementarymetal-oxide-semiconductor (CMOS) sensor may, at times, be insufficientto accurately represent outdoor scenes in a single image. This may beespecially true in the more compact sensors which may be used in mobiledevices, such as in the camera on a mobile telephone. For example, atypical sensor used in a mobile device camera may have a dynamic rangeof approximately 60-70 dB. However, a typical natural outdoor scene caneasily cover a contrast range of 100 dB between brighter areas and areaswith shadows. Because this dynamic range is greater than the dynamicrange of a typical sensor used in a mobile device, detail may be lost inimages captured by mobile devices.

One method which has been used to compensate for this lack of dynamicrange is to combine two or more frames into a single image with a higherdynamic range. For example, two of more frames with different exposurelengths may be combined into a single image. However, one problem withprevious techniques for combining multiple frames has been asignal-to-noise ratio discontinuity between frames of different exposurelengths. One method which may be used to demonstrate this problem is tocapture a grey ramp test chart using multiple exposures. In the portionof the grey ramp test chart corresponding to a transition point betweentwo successive frame exposures, higher levels of luma and chroma noisemay be observed. Such noise discontinuity may negatively affect imagequality.

SUMMARY

The systems, methods, devices, and computer program products discussedherein each have several aspects, no single one of which is solelyresponsible for its desirable attributes. Without limiting the scope ofthis invention as expressed by the claims which follow, some featuresare discussed briefly below. After considering this discussion, andparticularly after reading the section entitled “Detailed Description,”it will be understood how advantageous features of this inventioninclude robust estimation of color-dependent measurements.

In some aspects, a method of capturing a high dynamic range image isdescribed. The method includes capturing, on a first subset of pixels onan image sensor, a first image with a first exposure length. The methodfurther includes capturing, on a second subset of pixels on the imagesensor, a plurality of images with a second exposure length, wherein thesecond exposure length is shorter than the first exposure length. Themethod further includes combining the plurality of images with a secondexposure length into a second image and combining the first image andthe second image to form a high dynamic range image. In some aspects,the first subset of pixels may be a first plurality of pixel lines of apixel array and the second subset of pixels may be a second plurality ofpixel lines of a pixel array. In some aspects, the first plurality ofpixel lines and the second plurality of pixel lines may be interlacedwith each other. In some aspects, the plurality of images may comprisetwo or three images. In some aspects, the plurality of images may becaptures sequentially during the capture of the first image.

In some aspects, an electronic device for capturing a high dynamic rangeimage is disclosed. The device comprises a CMOS visible image sensorcomprising a plurality of pixels including a first subset of pixels anda second subset of pixels. The device further comprises a processorconfigured to capture, on the first subset of pixels, a first image witha first exposure length; capture, on a second subset of pixels, aplurality of images with a second exposure length, wherein the secondexposure length is shorter than the first exposure length; combine theplurality of images with a second exposure length into a second image;and combine the first image and the second image to form a high dynamicrange image.

In some aspects, an electronic device for capturing a high dynamic rangeimage is disclosed. The device comprises means for capturing, on a firstsubset of pixels on an image sensor, a first image with a first exposurelength; means for simultaneously capturing, on a second subset of pixelson an image sensor, a plurality of images with a second exposure length,wherein the second exposure length is shorter than the first exposurelength; means for combining the plurality of images with a secondexposure length into a second image; and means for combining the firstimage and the second image to form a high dynamic range image.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is an illustration of signal-to-noise ratio discontinuities whichmay occur when multiple frames are combined into a single image.

FIG. 2A is an illustration of a pixel which may be used to capture animage.

FIG. 2B is an illustration of buses which may be shared by a number ofpixels.

FIG. 2C is a timing diagram which may be used on an image sensor arrayin order to capture an image.

FIG. 2D is an illustration of a timing diagram for a rolling shutteroperation for a conventional image sensor.

FIG. 3A is a timing diagram which may be used on an image sensor arrayin order to capture an HDR image.

FIG. 3B is an illustration of a timing diagram for a rolling shutteroperation for an image sensor to capture an HDR image.

FIG. 4A is a timing diagram which may be used on an image sensor arrayin order to capture an HDR image using multiple short-exposure frames.

FIG. 4B is a timing diagram illustrating the timings for a rollingshutter operation for an image sensor to capture an HDR image usingmultiple short-exposure frames.

FIG. 5A is a timing diagram which may be used on an image sensor arrayin order to capture an HDR image using three short-exposure frames.

FIG. 5B illustrates a timing diagram for an image sensor array, whichcaptures three short-exposure frames per long-exposure frame.

FIG. 6A is an illustration of a proposed method of frame stackingmultiple short exposure frames to generate a higher fidelity frame.

FIG. 6B is an illustration of a proposed method of frame stacking threeshort exposure frames to generate a higher fidelity frame.

FIG. 7 illustrates the improvements in SNR which may be obtained byusing additional shorter-exposure frames.

FIG. 8 is an exemplary method of capturing an image.

FIG. 9 depicts a high-level block diagram of a device having a set ofcomponents including a processor operatively coupled to an image sensor.

DETAILED DESCRIPTION

Embodiments relate to systems, methods, and devices for using imagesensors to capture digital images with overlapped exposure bracketssupporting multiple short exposures. In one method, an image sensor maybe configured to simultaneously capture a first image with a relativelylonger exposure length, and two or more second images with relativelyshorter exposure lengths. These multiple second images with a shorterexposure length may be combined into a single shorter exposure lengthframe, which may then be combined with the first longer exposure image.Such a combination may allow for a high dynamic range (HDR) image, withsmaller signal-to-noise ratio (SNR) discontinuities than othertechniques. Such an imaging system may also allow for reduced amounts ofmotion blur compared to other techniques for capturing an HDR image.

FIG. 1 is a graph 100 of signal-to-noise ratio discontinuities which mayoccur when multiple frames are combined into a single image. Forexample, an image may be made up of a first frame 101, a second frame103, and a third frame 105. Each of these three frames may have adifferent exposure length.

For a given image sensor, there may be a maximum attainablesignal-to-noise ratio (SNR), SNRmax 110. For example, the best SNR maybe obtained as the image sensor reaches its well capacity. For any givenlight intensity, the single image may be based primarily on imageinformation received from the frame which has the highest SNR, but hasnot reached full-well capacity. For example, the first frame 101 mayhave the longest exposure length, and thus, the highest SNR at lowerlevels of light intensity, as the first frame 101 may be constructed bysensors which have received the most light. However, at higher levels oflight intensity, the first frame 101 may reach its full-well capacityand become saturated, and thus, details in such brighter areas may belost, and the readouts from the second frame 103 or third frame 105 maybe used instead. However, the SNR of each frame at a given lightintensity may be different than the SNR of the other frames at the samelight intensity, as each frame will have received differing levels oflight.

Accordingly, when three frames of different exposure lengths arecombined, there may be two SNR transition points. For example, attransition point 115, pixels with a light intensity less than this valuemay be based primarily on image information from the first frame 101,while pixels with a light intensity above this value may be based onimage information from the second frame 103. In some aspects, transitionpoint 115 may represent a level of light at which a frame with a longerexposure length, such as the first frame 101, reaches a full-wellcapacity. As illustrated in graph 100, at this transition point 115,there may be a large SNR discontinuity, such that pixels with a lightintensity of less than transition point 115 exhibit a high SNR, whilepixels with a light intensity of more than transition point 115 exhibita much lower SNR.

FIG. 2A is an illustration 230 of a pixel which may be used to capturean image. A pixel as illustrated in the illustration 230 may be referredto as a four-transistor (4T) pixel, as it has four transistors in eachpixel. In this pixel, incoming photons are converted to electron andhole pairs in a silicon substrate. The photo-electrons are thencollected by photodiode, Dph 240. In the beginning of integration, bothRST 248 and XFR 244 are high for a certain amount of time, turning onboth transistors Mrst 252 and MXFR 256. With CELLHI 260 being the supplyvoltage, this empties all electrons in photodiode Dph 240 and sets Dph240 to a predefined voltage. Once XFR 244 is low, MXFR 256 turns off andDph 240 starts collecting photo-electrons and its voltage goes down. Ingeneral, the rate of such photo-electron accumulation is proportional tothe amount of incoming light shining on the pixel. At the end ofintegration, typically a so-called correlated double sampling (CDS)operation is employed to read out the accumulated charge: first Mrst 252is turned on and off by setting RST 248 to a high value for a shorttime, which sets the floating node 264 (FN) to a reset voltage. The SEL268 signal is then set high, turning on Msel 272 to enable pixelreadout. In some aspects, SEL 268 may also be set high when RST 248 isset to a high value. If BUS 276 is connected to a current source, thenMsf 278 acts as a source follower, resulting in BUS 276 voltage trackingthe voltage of FN 264. Once the reset voltage of FN 264 has beenreadout, MXFR 256 is turned on, dumping all collected photo-electrons inDph 240 to FN 264, thus reducing the voltage of FN 264. After this, BUS276 voltage will follow the reduced voltage of FN 264 and a secondreadout is performed through the source follower. The difference betweenthe two readouts gives the voltage change on node FN 264 due tophoto-electrons collected by Dph 240. CDS operation in general reducescertain temporal noises and the impact of transistor variation. The timedifference between two XFR 244 pulses, one for reset and one forreadout, is the integration time of the pixel.

FIG. 2B is an illustration 280 of buses which may be shared by a numberof pixels. Generally, for a pixel array which contains a large number,often millions, of pixels, it is not possible to connect to all of thedifferent control signals in FIG. 2A individually. Instead, SEL 285, RST286, and XFR 287 are horizontal buses which may be shared by one row ofpixels, while BUS is a vertical bus shared by one column of pixels. Inthe illustration 280, CELLHI (such as CELLHI 260 of FIG. 2A) itself maybe the same for each pixel in the array, and therefore can be either ahorizontal or a vertical bus. Because of this implementation wherein SEL285, RST 286, and XFR 287 are horizontal buses, each pixel in a row mayhave the same integration time.

FIG. 2C is a timing diagram 200 which may be used on an image sensorarray in order to capture an image. For example, the image sensor arraymay contain a number of lines, numbered i, i+1, i+2, and so on. At atime 210, image sensors on the first line i 205 may be reset, and thusmay begin to collect incoming light. After an integration time T 220, ata time 211, image sensors on the first line i 205 may be read. That is,at time 211, it may be determined how much light image sensors on thefirst line i 205 have received since their last reset, at time 210. Theinformation captured from the image sensors on the first line i 205between time 210 and time 211 may be used as part of a first frame.After this, the image sensors on the first line i 205 may be reset attime 212, in order to begin a capture for a subsequent frame. Asillustrated in timing diagram 200, each line may be similarly reset andread, with each readout occurring after the same integration time T 220on each given line. Accordingly, timings based upon this timing diagrammay be used to capture a number of frames, each with the same exposurelength.

However, one disadvantage of such a timing is that it may only be ableto capture images with a single exposure length. Thus, it may not bepossible to use such a timing diagram for an HDR image. In order tocreate an HDR image, a readout based on timing diagram 200 may be used,in which a longer-exposure frame is taken for the first frame, and arelatively shorter exposure frame is taken as a second frame. Theseframes may then be combined. However, one drawback of such an approachis that the two frames will be taken at different times, rather thansimultaneously. Thus, if a subject of the photograph is moving, this maycreate a blurry image or ghosting, as the moving subject may be in adifferent location in the longer- and the shorter-exposure frames.

FIG. 2D is an illustration 1000 of a timing diagram for a rollingshutter operation for a conventional image sensor. This illustration1000 shows the timings of various pulses, when a sensor is used asillustrated in illustration 200 of FIG. 2C. This illustration 1000illustrates the timings which may be used to an array with M rows ofpixels. First, at time zero 1048, integration of the first row may beginby turning on both RST1 and XFR1 for a short time. Because of this, allphotodiodes in the first row will be set to the pinning voltage of thephotodiode. At time Tline 1050, a similar sequence is repeated to resetrow 2, followed by row 3 at time 2*Tline, and so on. This continuesuntil last row, row M, is reset at time (M−1)*Tline 1056. Assuming anintegration time of T1, at time T1 1052, row 1 will be readout. Thisinvolves turning on & off RST1 1010 to reset floating nodes, turning onSEL1 1014 to readout reset level, turning on & off XFR1 1018 to dumpcharges from photodiodes to floating nodes, and then reading out signallevel to perform CDS. One line time later, at T1+Tline 1054, row 2 willbe read out using the same procedure, as described above. This continuesuntil each row is read out. Since each pixel has two operation sequencesduring the capture of an image: reset and readout, we can think of eachpicture frame as consisting of two different frames: a reset frame and areadout frame. For the frame discussed above, frame N, its reset framestarts at time zero 1048 and ends at time M*Tline and its readout framestarts at time T1 1052 and ends at T1+M*Tline. Once row 1 has finishedreadout of frame N, it can now be reset to start integration of frameN+1. The time at which frame N+1 begins to be reset may depend on theframe time, Tframe 1058, and whether the integration time for a framemay change between frame N and frame N+1. If the integration time offrame N+1 is the same as that of frame N, then frame N+1 may be reset attime Tframe 1058, and the sequence described above may be repeated.Based on this diagram, it may be observed that reset frames will notoverlap each other, and readout frames will not overlap each other.However, a reset frame may overlap the previous readout frame.

FIG. 3A is a timing diagram 300 which may be used on an image sensorarray in order to capture an HDR image. In timing diagram 300, theeven-numbered lines act just as all the lines in timing diagram 200 did.That is, each of these lines resets 310 and reads 311 after anintegration time T1 330, just as each line in illustration 200 of FIG.2C reset 210 and read 211 after an integration time T.

However, in timing diagram 300, the odd-numbered lines may be reset 320and may be read 321 after integration time T2 331. Integration time T2331 may be a shorter time than integration time T1 330. Accordingly, theeven-numbered lines may be combined to construct an image with anintegration time T1 330. Similarly, the odd-numbered lines may becombined to construct an image with an integration time T2 331. Thesetwo images may be combined into a single image in order to form an HDRimage.

This approach may offer both advantages and disadvantages over theapproach in timing diagram 200 of FIG. 2C. For example, this approachmay allow for the construction of an HDR image. Additionally, unlike anHDR image that might be constructed using sequential longer- andshorter-exposure frames, the two frames in an HDR image formed using thetiming diagram 300 are captured almost simultaneously. One advantage ofthe two frames being captured simultaneously is that moving objects maybe in more similar locations in both the shorter-exposure and thelonger-exposure image. This may reduce blurriness in the combinedimages, and may also reduce ghosting in the combined image. Onetrade-off for the timing diagram 300 is that the vertical resolution ofsuch a combined image may be half the vertical resolution of an imagecaptures using timing diagram 200. For example, each of the longer- andthe shorter-exposure images may use only half of the lines of the array.Thus, the vertical resolution of each of the images may be half as muchas an image that uses the full array. However, such a loss of resolutionmay be less of a concern as image sensors include a larger number oflines, as such a loss of vertical resolution may be seen as a worthwhiletradeoff, in order to capture HDR images with less blurriness andghosting. The shorter-exposure frames may have an exposure length thatis based on the exposure length of the longer-exposure frames. Forexample, a shorter-exposure frame may have an exposure length that isapproximately 10%, 20%, 25%, 33%, 50% or some other proportion of theexposure length of the longer-exposure frame. In some aspects, theexposure length of the longer-exposure frame may be based upon theamount of light that is present in a scene. For example, the exposurelength of the longer-exposure frame may be configured to be a length inwhich a certain proportion of the pixels of the capture device reachfull-well capacity. For example, this proportion may be 5% of thepixels, 10%, 20%, or another proportion of the pixels of the capturedevice. In some aspects, the length of the shorter-exposure frame may bebased on the length of the longer-exposure frame, based on the amount oflight that is present in a scene, or based on other factors.

FIG. 3B is an illustration 1100 of a timing diagram for a rollingshutter operation for an image sensor to capture an HDR image. Forexample, this timing diagram may correspond to a timing embodiment suchas that illustrated timing diagram 300 of FIG. 3A. As described above,one technique to increase dynamic range of an image is to use twodifferent integration times for the odd rows and for the even rows of animage. For example, the odd rows may have a longer integration time, T1,allowing them to show more detail in dark areas, while the even rows mayhave a shorter integration time, T2, which allows them to show moredetail in bright areas. For video mode applications, it may bebeneficial to do so inside the same frame. In order to simplify thisdiagram, each row does not display individual signals, but instead showsonly a “read” and a “reset” pulse. At time zero 1160, the first line isreset by doing the RESET1 1110 operation. At time 2*Tline 1162, line 3will be reset using a RESET3 1126 operation. At time Tline, aconventional imager would reset line 2. However, in this imager, line 2is a shorter-exposure line, and thus does not need to be reset untillater. However, line 2 is not reset until time T1-T2+Tline 1164, using aRESET2 1114 operation. This timing is determined in order to allow eachline, including those with exposure time T1 and those with exposure timeT2, to be read out in evenly-spaced intervals of Tline, as inillustration 1100. For example, line 1 will be readout using a READ11114 operation at time T1 1166. Following this, line 2 will be readoutusing a READ2 1122 operation at time T1+Tline 1168. This procedure willcontinue until the last line of the M line array.

In illustration 1100, it may be observed that while the readout frameremains the same as in illustration 1000 of FIG. 2D, with each row beingread out sequentially, the reset frame is split into two differentframes—one for the odd rows and one for the even rows. As inillustration 1000, there is no overlap between consecutive readoutframes. However, there may be overlap between reset frames and readoutframes, or overlap between even- and odd-row reset frames. Indeed,pixels from two different rows, such as an even row and an odd row, maybe reset at the same time. Such simultaneous resets will require thatthe common CELLHI bus has enough driving capability. However, this maynot be problematic, as the reset of a pixel row many not consume muchpower. This illustration 1100 may work for a black-and-white pixel, andmay easily be adapted for a color array. For example, in a color imagesensor with a 2×2 Bayer CFA pattern, each row in illustration 1100 maybe expanded to two rows, in order to match the Bayer pattern.Accordingly, there would be reset frames for odd pairs of rows anddifferent reset frames for even pairs of rows.

In some aspects, however, it may be beneficial to use multipleshort-exposure frames for each long-exposure frame. The multipleshort-exposure frames may be combined together, and this combined framemay increase the SNR of the combined frame. Accordingly, a pixel arrayreadout scheme which leads to multiple short exposure frames for eachand every long-exposure frame may be beneficial. These short-exposureframes may be read out from the sensor, and then averaged to generate ahigh-fidelity short exposure frame.

FIG. 4A is a timing diagram 400 which may be used on an image sensorarray in order to capture an HDR image using multiple short-exposureframes. Timing diagram 400 is similar in a number of respects to timingdiagram 300, however, one important difference is that timing diagram400 captures multiple short-exposure frames during a singlelong-exposure frame. As in timing diagram 300 of FIG. 3A, alonger-exposure frame may be captured on the even-numbered lines of anarray of pixels. A reset 410 may occur on the first line, followed by aread 411 operation, after the integration time T1 430. However, while intiming diagram 300 a single short-exposure frame was taken during theintegration time T1 430, in timing diagram 400, two short-exposureframes may be taken during the integration time T1 430.

For example, on line i+1, a reset 420 may occur followed by a read 421after integration time T2 431. After this, another reset 422 may befollowed by another read 423. The timing may be followed on eachodd-numbered line. These two shorter-exposure frames may then becombined together, such as by averaging or using other techniques,including techniques for minimizing motion blur. The combinedshorter-exposure frame may then be combined with the longer-exposureframe which is captured simultaneously on the even-numbered lines. Byusing two or more short-exposure frames in the combined HDR image, SNRdiscontinuity may be reduced, as the combined shorter-exposure frame mayhave a higher SNR than each of the individual shorter-exposure frames.Further, by capturing both of the shorter-exposure frames simultaneouslywith capturing the longer-exposure frame, motion blur and ghosting inthe combined image may be reduced. As with timing diagram 300, thisapproach may reduce vertical resolution in the combined image. However,with a high-resolution image sensor, this loss of resolution may not beproblematic.

Note that while timing diagram 400 illustrates these even-numbered linesmay use integration time T1, and odd-numbered lines use integration timeT2, other schemes may also be used to divide the pixels which useintegration time T1 and those which use integration time T2. Forexample, in some aspects, the display may be configured such that halfof the pixels in line i use integration time T1, and half of the pixelsin line i use integration time T2. For example, while the pixels in FIG.4A are divided based upon even and odd line numbers, other divisions mayalso be possible. In some aspects, it may be beneficial to insteaddivide the pixels of a capture device into groups based upon whichcolumn they are in. In some aspects, it may be beneficial to divide thepixels between the longer- and the shorter-integration time evenlyacross rows and columns. For example, the pixels may be divided into twointegration times based upon a checkerboard pattern. Such a division maybe advantageous, as this may result in a resolution loss in an HDR imagethat is more uniform in the x & y directions.

FIG. 4B is another timing diagram 450, illustrating the timings for arolling shutter operation for an image sensor to capture an HDR imageusing multiple short-exposure frames. In this scheme, the odd rows areintegrated with longer-integration time T1. However, during this time,the even rows capture two frames with shorter-integration time T2.Accordingly, there are two readout and two reset frames on even rows forevery one reset and readout frame on the odd rows.

At time zero 470, frame N is reset by resetting row 1 using RESET1 455,followed by resetting row 2 using RESET2 257 at Tline 471, and so on,until each of the M rows have been reset. This is one full reset frame,which may be split into two reset frames (one odd and one even), ifdesired. For example, this may be referred to as both the odd row resetframe N, and the 1^(st) even row reset frame N. Note that the 1^(st)even row reset frame N does not need to be started at time Tline 471.The 1^(st) even row reset frame N may be started at any time during theintegration time of the first long-exposure frame.

After the short integration time T2 after the reset of row 2, at timeT2+Tline 472, a READ2 458 operation will be done one row 2, and then foreach even row after this in order. This may be referred to as the 1^(st)even row readout. Accordingly, the first of the shorter-exposure framesmay be captured in this way.

At time T1-T2+Tline 473, the 2^(nd) even row reset frame may be started.As with the 1^(st) even row reset, this process begins by resetting row2, and then row 4, and so on. At time T1, a complete readout frame maybegin, consisting of the odd row readout frame and 2^(nd) even rowreadout frame. These odd and even readout frames are staggered by oneline time delay, thus at any line time, there is only one row beingreadout for this frame. Note that while the odd row readout frame andthe 2^(nd) even row readout frame are shown to be inter-coupled, andcompleted at the same time, these do not need to be done at the sametime. For example, just as the 1^(st) even row reset frame need not becoupled to the odd row reset frame, neither do the readout frames needto be coupled together as long as they are separated by an odd number ofline times so that their readout won't occur at the same time. This willallow us to fully utilize the whole video frame time for T2. The onlyconstraint regarding the timing of the 2^(nd) even row reset frame N isthat is must begin after the 1^(st) even row readout frame N begins, andthe 2^(nd) even row readout frame N must begin prior to the beginning ofthe 1^(st) even row reset frame N+1. If these conditions were not met,then a single row would be scheduled to be reset for a next frame priorto that row being read for the previous frame, which would erase theintegrated signal from previous frame and result in the wrong readout.The illustrated sequence of illustration 450 may be repeated for eachframe, such as a frame of video. In order to read out additional evenrow readout frames, faster analog readout chains need to be implemented,together with faster data output interface.

In some aspects, more than two shorter-exposure frames may be takensimultaneously with the longer-exposure frame. For example, FIG. 5A is atiming diagram 500 which may be used on an image sensor array in orderto capture an HDR image using three short-exposure frames. In timingdiagram 500, while line i 501 captures a single frame with anintegration time T1 530, line i+1 502 captures three shorter-exposureframes, each with an integration time T2 531. Capturing moreshorter-exposure frames may help increase the SNR in a frame made up ofthe combined shorter-exposure frames. For example, combining threeshorter-exposure frames into a single image may result in a larger SNRthan combining two shorter-exposure frames with the same exposurelength. Accordingly, when such a combined image is combined with thelonger-exposure frame captured on the even-numbered lines such as line i501, SNR discontinuities in the formed image may be reduced.

FIG. 5B illustrates a timing diagram 550 for an image sensor array,which captures three short-exposure frames per long-exposure frame.Timing diagram 550 is similar to timing diagram 450. However, timingdiagram 550 includes an additional even row reset frame and even rowreadout frame during the time of the odd row reset frame. Because wecan't readout two different rows at the same time, the exact readoutsequence must be set up to avoid this. For example, if the odd rowreadout occurs at first half of each line time, we can have the last(3^(rd)) even row readout occurring also at first half of each line timebecause they are staggered by one line time delay. For example, thefirst row may be read out at time T1 560, while the second row may beread out at time T1+Tline 561, and so on. With that, the first two evenreadout rows may occurs at second half of each line time so they won'tconflict with the odd row readout frame or the 3^(rd) even row readoutframe. This means that there will be a constraint on Tmid 558, which isthe spacing between the 1^(st) and 2^(nd) even row readout frames—Tmid558 must be in such a way that these two frames are also staggered byone line time. Accordingly, Tmid 558 must be an even multiple of Tline556, as this will prevent two lines from 1^(st) and 2^(nd) even readoutframes from being read out at the same time, and will stagger linereadouts by one line time, Tline 556.

While the timing described above has 1^(st) even row reset frame tied tothe odd row reset frame and 3^(rd) even row readout frame tied to theodd row readout frame, following the same discussion in the previouscase with two even row readout frames, we can decouple such relationshipand move the even row readout/reset frame around. The only constrain isthat at any time, there can only be one row being readout and thebeginning of next even row reset frame cannot be ahead of the currenteven row readout frame.

FIG. 6A is an illustration of one method of frame stacking usingmultiple short exposure frames to generate a higher fidelity frame. Thismethod may be done by a device for capturing digital images, such as acellular telephone or a digital camera. This method may be used inconjunction with timing diagram 400 of FIG. 4A. At block 602, an imagesensor supporting temporally overlapped shorter-exposure frames maycapture three frames including two frames 605 a, 605 b with an exposuretime of T2, and one frame 604 with a longer exposure time of T1. Thesethree frames may be transmitted via a higher bandwidth readout comparedto other image sensors. These three frames may be captured using animage sensor timing like that illustrated in timing diagram 400.

At block 606, the two shorter-exposure frames 605 a, 605 b with exposurelength T2, may be combined using frame averaging 606. In some aspects,this frame averaging 606 may include taking a simple average of thepixel values of each pixel in the two frames 605 a, 605 b. In someaspects, frame averaging may also include certain techniques for motioncompensation, such as techniques which might detect motion between thetwo frames and might attempt to produce a combined image with lessblurriness or ghosting. In some aspects, the frame averaging 606 mayinclude using a weighted average. Accordingly, the two shorter-exposureframes 605 a, 605 b may be combined into a single high fidelity frame608, still with an exposure length T2. Because the high fidelity frame608 is made up of two frames 605 a, 605 b which have been combinedtogether, such as by frame averaging 606, the high fidelity frame 608may have less noise than each of the two shorter-exposure frames 605 a,605 b.

Finally, a high fidelity HDR frame 612 may be created by combining thehigh fidelity frame 608 with the longer-exposure frame 609. Thiscombination may comprise linearizing and blinding 610 the high fidelityframe 608 with the longer-exposure frame 609. This HDR frame 612 mayhave a higher dynamic range than either the high fidelity frame 608 orthe longer-exposure frame 609. For example, the high fidelity frame 608may, due to its shorter-exposure length T2, lack certain details indimly-lit areas of the image. Similarly, the longer-exposure frame 609may lack certain details in brightly-lit areas of the image, as thoseareas may have reached their full-well capacity during the longerexposure length T1. However, the high fidelity HDR frame 612 may includethese details in both the bright and the dim areas of the photo.Additionally, because of the higher SNR of the high fidelity frame 608,compared to a single frame with an exposure length T2, the high fidelityHDR frame 612 may have smaller SNR discontinuities than other types ofHDR frame.

FIG. 6B is an illustration of a proposed method of frame stacking usingthree short exposure frames to generate a higher fidelity frame. Thismethod may be done by a device for capturing digital images, such as acellular telephone or a digital camera. This method may be used inconjunction with the timing diagram 500 illustrated in FIG. 5A. Thedifference between the method illustrated in FIG. 6A and that of FIG. 6Bis that the method of FIG. 6B uses three shorter-exposure frames 605 c,605 d, 605 e, rather than two such frames. The use of additionalshorter-exposure frames may further increase the SNR of thehigh-fidelity frame 608 a. In some aspects, any number ofshorter-exposure frames may be combined to form a high fidelity frame608 a. In some aspects, the more shorter-exposure frames that are used,the higher the SNR of the high fidelity frame 608 a will be.

FIG. 7 illustrates the improvements in SNR which may be obtained byusing additional shorter-exposure frames. For example, graph 700illustrates an improvement in SNR from using a single shorter-exposureframe with an exposure of T2, represented by line 710, and the SNR of aframe which is constructed by stacking multiple frames with an exposurelength T2, represented by line 720. As graph 700 shows, stackingadditional frames with the same exposure length T2 will increase the SNRof the stacked image, as compared to an individual frame. Similarly,stacking an increasing number of frames will continue to improve SNR.However, note that adding additional frames may also increase thelikelihood that moving objects may appear blurry or may exhibitghosting. Accordingly, in some aspects, it may be desirable to stack anumber of shorter-exposure frames, where the number is chosen based uponminimizing ghosting while maximizing SNR. For example, it may bedesirable to stack a number of shorter-exposure frames which are alltaken during the time which the longer-exposure frame is taken, such asusing timings as illustrated in timing diagram 400 of FIG. 4A or timingdiagram 500 of FIG. 5A.

FIG. 8 is an exemplary method 800 of capturing an HDR image. This methodmay be done by a digital device, such as a digital camera, a cell phone,or another device which includes an image sensor.

At block 802, the method 800 determines a first exposure length. Forexample, this exposure length may be determined based upon user input,such as a user selecting an exposure length using an interface of thedigital device. In some aspects, the first exposure length may bedetermined based, at least in part, on a light level which is found in ascene to be captured. For example, the first exposure length may be anexposure length that is configured to allow a certain proportion of thepixels of the imaging device to reach a full-range output, such as 5%,10%, 20% or another proportion of the pixels. In some aspects, the firstexposure length may be based upon an amount of movement that is detectedin a given scene. The first exposure length may be based on any of theabove factors, alone or in combination with each other and with otherfactors.

At block 805, the method 800 captures a first image with the firstexposure length. This image is captured using a first subset of pixelson an image sensor. For example, the first subset of pixels may be asubset of the lines of pixels, such as the odd-numbered rows of pixels.In some aspects, the pixels may be divided into two or more subsets ofpixels, such as dividing the pixels into even rows and odd rows.

At block 807, the method 800 determines a second exposure length that isshorter than the first exposure length. For example, the second exposurelength may be based on the light level in the scene to be captured. Thesecond exposure length may also be based on the first exposure length.For example, the second exposure length may be based upon someproportion of the first exposure length, such as being approximately10%, 20%, 25%, 33%, 50% or some other proportion of the first exposurelength. In some aspects, it may be beneficial to base the secondexposure length on the first exposure length, in order to allow thedigital device to capture a particular number of shorter-exposure frameswith the second exposure length during the capture of a singlelonger-exposure frame with the first exposure length.

At block 810, the method 800 then captures a plurality of images with asecond exposure length on a second subset of pixels on an image sensor,wherein the second exposure length is shorter than the first exposurelength, and wherein the plurality of images are captured simultaneouslywith the capture of the first image. For example, while the first subsetof pixels is capturing a first image, the second subset of pixels maycapture two or more images. For example, the second subset of pixels maybe the even-numbered lines of an array of pixels on an image sensor. Thesecond subset of pixels, in some aspects, may capture, two, three, ormore images during the time the first subset of pixels captures animage. Each of these images may be captured sequentially on the secondsubset of pixels.

At block 815, the method 800 combines the plurality of images with asecond exposure length into a second image. For example, thiscombination may be made by using an average, or a weighted averagebetween the images. This combined image may have less noise that eachindividual image of the plurality of images. At block 820, the method800 combines the first image and the second image. This combined imagemay be constructed using pixel averaging, using a weighted pixelaverage, or other image combination techniques. In some aspects, thecombined image may by an HDR image, with a higher dynamic range thaneither an image with the first exposure length, or an image with thesecond exposure length.

FIG. 9 depicts a high-level block diagram of a device 900 having a setof components including a processor 920 operatively coupled to an imagesensor 915. A working memory 905, storage 910, and memory 930 are alsoin communication with and operative attached to the processor. Device900 may be a device configured to take digital photographs and/orvideos, such as a digital camera, a cell phone, or another device. Theimage sensor 915 may be configured to capture a number of pixels of animage. Any number of such pixels may be included on the device 900. Thepixels on an image sensor may be arranged in a number of rows andcolumns, in order to form a grid.

Processor 920 may be a general purpose processing unit or a processorspecially designed for the disclosed methods. As shown, the processor920 is connected to a memory 930 and a working memory 905. In theillustrated embodiment, the memory 930 stores image capture module 935,image combination module 940, and operating system 945. These modulesinclude instructions that configure the processor to perform varioustasks. Working memory 905 may be used by processor 920 to store aworking set of processor instructions contained in the modules of memory930. Alternatively, working memory 905 may also be used by processor 920to store dynamic data created during the operation of device 900.

As mentioned above, the processor 920 is configured by several modulesstored in the memories. For example, the image capture module 935 mayinclude instructions that configure the processor 920 to capture one ormore images using the image sensor 915. For example, the image capturemodule 935 may include instructions that configure the processor 920 tocapture a first image with a first subset of pixels in image sensor 915,while simultaneously capturing a plurality of images with a secondsubset of pixels in image sensor 915. For example, this may allow afirst image to be captured on the first subset of pixels using a firstexposure length, while a plurality of images with a second exposurelength, shorter than the first exposure length, are captured on thesecond subset pixels of image sensor 915.

The memory 930 may also contain an image combination module 940. Theimage combination module 940 may contain instructions that configure theprocessor 920 to receive signals from the image sensor 915, and combinea number of frames from the image sensor 915 into a single frame. Insome aspects, the image combination module 940 may be configure tooperate in parallel with the image capture module 935, in order tocombine frames captured using an image capture process as describedabove.

Operating system module 945 configures the processor to manage thememory and processing resources of device 900. For example, operatingsystem module 945 may include device drivers to manage hardwareresources such as the image sensor 915 or storage 910. Therefore, insome embodiments, instructions contained in modules discussed above maynot interact with these hardware resources directly, but insteadinteract through standard subroutines or APIs located in operatingsystem component 945. Instructions within operating system 945 may theninteract directly with these hardware components.

Processor 920 may write data to storage module 910. While storage module910 is represented graphically as a traditional disk device, those withskill in the art would understand multiple embodiments could includeeither a disk based storage device or one of several other type storagemediums to include a memory disk, USB drive, flash drive, remotelyconnected storage medium, virtual disk driver, or the like.

FIG. 9 depicts a device having separate components to include aprocessor, and memory, one skilled in the art would recognize that theseseparate components may be combined in a variety of ways to achieveparticular design objectives. For example, in an alternative embodiment,the memory components may be combined with processor components to savecost and improve performance.

Additionally, although FIG. 9 illustrates two memory components, toinclude memory component 930 having several modules, and a separatememory 905 having a working memory, one with skill in the art wouldrecognize several embodiments utilizing different memory architectures.For example, a design may utilize ROM or static RAM memory for thestorage of processor instructions implementing the modules contained inmemory 930. Alternatively, processor instructions may be read at systemstartup from a disk storage device that is integrated into device 900 orconnected via an external device port. The processor instructions maythen be loaded into RAM to facilitate execution by the processor. Forexample, working memory 905 may be a RAM memory, with instructionsloaded into working memory 905 before execution by the processor 920.

It should be understood that any reference to an element herein using adesignation such as “first,” “second,” and so forth does not generallylimit the quantity or order of those elements. Rather, thesedesignations may be used herein as a convenient method of distinguishingbetween two or more elements or instances of an element. Thus, areference to first and second elements does not mean that only twoelements may be employed there or that the first element must precedethe second element in some manner. Also, unless stated otherwise a setof elements may include one or more elements.

A person/one having ordinary skill in the art would understand thatinformation and signals may be represented using any of a variety ofdifferent technologies and techniques. For example, data, instructions,commands, information, signals, bits, symbols, and chips that may bereferenced throughout the above description may be represented byvoltages, currents, electromagnetic waves, magnetic fields or particles,optical fields or particles, or any combination thereof.

A person/one having ordinary skill in the art would further appreciatethat any of the various illustrative logical blocks, modules,processors, means, circuits, and algorithm steps described in connectionwith the aspects disclosed herein may be implemented as electronichardware (e.g., a digital implementation, an analog implementation, or acombination of the two, which may be designed using source coding orsome other technique), various forms of program or design codeincorporating instructions (which may be referred to herein, forconvenience, as “software” or a “software module), or combinations ofboth. To clearly illustrate this interchangeability of hardware andsoftware, various illustrative components, blocks, modules, circuits,and steps have been described above generally in terms of theirfunctionality. Whether such functionality is implemented as hardware orsoftware depends upon the particular application and design constraintsimposed on the overall system. Skilled artisans may implement thedescribed functionality in varying ways for each particular application,but such implementation decisions should not be interpreted as causing adeparture from the scope of the present disclosure.

The various illustrative logical blocks, modules, and circuits describedin connection with the aspects disclosed herein and in connection withFIGS. 1-9 may be implemented within or performed by an integratedcircuit (IC), an access terminal, or an access point. The IC may includea general purpose processor, a digital signal processor (DSP), anapplication specific integrated circuit (ASIC), a field programmablegate array (FPGA) or other programmable logic device, discrete gate ortransistor logic, discrete hardware components, electrical components,optical components, mechanical components, or any combination thereofdesigned to perform the functions described herein, and may executecodes or instructions that reside within the IC, outside of the IC, orboth. The logical blocks, modules, and circuits may include antennasand/or transceivers to communicate with various components within thenetwork or within the device. A general purpose processor may be amicroprocessor, but in the alternative, the processor may be anyconventional processor, controller, microcontroller, or state machine. Aprocessor may also be implemented as a combination of computing devices,e.g., a combination of a DSP and a microprocessor, a plurality ofmicroprocessors, one or more microprocessors in conjunction with a DSPcore, or any other such configuration. The functionality of the modulesmay be implemented in some other manner as taught herein. Thefunctionality described herein (e.g., with regard to one or more of theaccompanying figures) may correspond in some aspects to similarlydesignated “means for” functionality in the appended claims.

If implemented in software, the functions may be stored on ortransmitted over as one or more instructions or code on acomputer-readable medium. The steps of a method or algorithm disclosedherein may be implemented in a processor-executable software modulewhich may reside on a computer-readable medium. Computer-readable mediaincludes both computer storage media and communication media includingany medium that can be enabled to transfer a computer program from oneplace to another. A storage media may be any available media that may beaccessed by a computer. By way of example, and not limitation, suchcomputer-readable media may include RAM, ROM, EEPROM, CD-ROM or otheroptical disk storage, magnetic disk storage or other magnetic storagedevices, or any other medium that may be used to store desired programcode in the form of instructions or data structures and that may beaccessed by a computer. Also, any connection can be properly termed acomputer-readable medium. Disk and disc, as used herein, includescompact disc (CD), laser disc, optical disc, digital versatile disc(DVD), floppy disk, and blu-ray disc where disks usually reproduce datamagnetically, while discs reproduce data optically with lasers.Combinations of the above should also be included within the scope ofcomputer-readable media. Additionally, the operations of a method oralgorithm may reside as one or any combination or set of codes andinstructions on a machine readable medium and computer-readable medium,which may be incorporated into a computer program product.

It is understood that any specific order or hierarchy of steps in anydisclosed process is an example of a sample approach. Based upon designpreferences, it is understood that the specific order or hierarchy ofsteps in the processes may be rearranged while remaining within thescope of the present disclosure. The accompanying method claims presentelements of the various steps in a sample order, and are not meant to belimited to the specific order or hierarchy presented.

Various modifications to the implementations described in thisdisclosure may be readily apparent to those skilled in the art, and thegeneric principles defined herein may be applied to otherimplementations without departing from the spirit or scope of thisdisclosure. Thus, the disclosure is not intended to be limited to theimplementations shown herein, but is to be accorded the widest scopeconsistent with the claims, the principles and the novel featuresdisclosed herein. The word “exemplary” is used exclusively herein tomean “serving as an example, instance, or illustration.” Anyimplementation described herein as “exemplary” is not necessarily to beconstrued as preferred or advantageous over other implementations.

Certain features that are described in this specification in the contextof separate implementations also can be implemented in combination in asingle implementation. Conversely, various features that are describedin the context of a single implementation also can be implemented inmultiple implementations separately or in any suitable sub-combination.Moreover, although features may be described above as acting in certaincombinations and even initially claimed as such, one or more featuresfrom a claimed combination can in some cases be excised from thecombination, and the claimed combination may be directed to asub-combination or variation of a sub-combination.

Similarly, while operations are depicted in the drawings in a particularorder, this should not be understood as requiring that such operationsbe performed in the particular order shown or in sequential order, orthat all illustrated operations be performed, to achieve desirableresults. In certain circumstances, multitasking and parallel processingmay be advantageous. Moreover, the separation of various systemcomponents in the implementations described above should not beunderstood as requiring such separation in all implementations, and itshould be understood that the described program components and systemscan generally be integrated together in a single software product orpackaged into multiple software products. Additionally, otherimplementations are within the scope of the following claims. In somecases, the actions recited in the claims can be performed in a differentorder and still achieve desirable results.

What is claimed is:
 1. A method of capturing a high dynamic range image,comprising: capturing, on a first subset of pixels on an image sensor, afirst image of a scene with a first exposure length; determining asecond exposure length based on a rate of photo-electron accumulation onone or more pixels used to capture the first image of the scene, whereinthe second exposure length is shorter than the first exposure length;capturing, on a second subset of pixels on the image sensor, a secondimage of the scene with the second exposure length and third image ofthe scene with the second exposure length, wherein the second image andthe third image are captured sequentially during the capturing of thefirst image; and combining the first image, the second image, and thethird image to form a high dynamic range image of the scene.
 2. Themethod of claim 1, further comprising setting the first exposure lengthto be a length at which a certain proportion of pixels reach full-wellcapacity.
 3. The method of claim 1, wherein the first subset of pixelscomprises a first plurality of pixel lines of a pixel array and whereinthe second subset of pixels comprises a second plurality of pixel linesof the pixel array.
 4. The method of claim 3, wherein the firstplurality of pixel lines and the second plurality of pixel lines areinterlaced with each other.
 5. The method of claim 1, wherein the firstimage and the second image each comprise at least one readout frame andone reset frame.
 6. The method of claim 1, wherein the rate ofphoto-electron accumulation is based on employing correlated doublesampling (CDS) to read out accumulate charge on the one or more pixelsused to capture the first image of the scene.
 7. An electronic devicefor capturing a high dynamic range image, comprising: a CMOS visibleimage sensor comprising a plurality of pixels including a first subsetof pixels and a second subset of pixels; and a processor coupled to theimage sensor and configured to: capture, on the first subset of pixelsof the image sensor, a first image of a scene with a first exposurelength; determine a second exposure length based on a rate ofphoto-electron accumulation on one or more pixels used to capture thefirst image of the scene, wherein the second exposure length is shorterthan the first exposure length; capture, on a second subset of pixels onthe image sensor, a second image of the scene with the second exposurelength and third image of the scene with the second exposure length,wherein the second image and the third image are captured sequentiallyduring the capturing of the first image; and combine the first image,the second image, and the third image to form a high dynamic range imageof the scene.
 8. The electronic device of claim 7, wherein the processoris further configured to set the first exposure length to be a length atwhich a certain proportion of pixels reach full-well capacity.
 9. Theelectronic device of claim 7, wherein the first subset of pixelscomprises a first plurality of pixel lines of a pixel array and whereinthe second subset of pixels comprises a second plurality of pixel linesof the pixel array.
 10. The electronic device of claim 9, wherein thefirst plurality of pixel lines and the second plurality of pixel linesare interlaced with each other.
 11. The electronic device of claim 7,wherein the first image and the second image each comprise at least onereadout frame and one reset frame.
 12. The electronic device of claim 7,the rate of photo-electron accumulation is based on employing correlateddouble sampling (CDS) to read out accumulate charge on the one or morepixels used to capture the first image of the scene.
 13. An apparatusfor capturing a high dynamic range image, comprising: means forcapturing, on a first subset of pixels on an image sensor, a first imageof a scene with a first exposure length; means for determining a secondexposure length based on a rate of photo-electron accumulation on one ormore pixels used to capture the first image of the scene, wherein thesecond exposure length is shorter than the first exposure length; meansfor capturing, on a second subset of pixels on the image sensor, asecond image of the scene with the second exposure length and thirdimage of the scene with the second exposure length, wherein the secondimage and the third image are captured sequentially during the capturingof the first image; and means for combining the first image, the secondimage, and the third image to form a high dynamic range image of thescene.
 14. The apparatus of claim 13, further comprising means forsetting the first exposure length to be a length at which a certainproportion of pixels reach full-well capacity.
 15. The apparatus ofclaim 13, wherein the first subset of pixels comprises a first pluralityof pixel lines of a pixel array and wherein the second subset of pixelscomprises a second plurality of pixel lines of the pixel array.
 16. Theapparatus of claim 15, wherein the first plurality of pixel lines andthe second plurality of pixel lines are interlaced with each other. 17.The apparatus of claim 13, wherein the second image comprises a combinedplurality of images with the second exposure length.
 18. The apparatusof claim 13, wherein the first image and the second image each compriseat least one readout frame and one reset frame.
 19. The apparatus ofclaim 13, wherein the rate of photo-electron accumulation is based onemploying correlated double sampling (CDS) to read out accumulate chargeon the one or more pixels used to capture the first image of the scene.20. A non-transitory computer readable storage medium having storedthereon instructions that, when executed, cause a processor of a deviceto: capture, on the first subset of pixels of the image sensor, a firstimage of a scene with a first exposure length; determine a secondexposure length based on a rate of photo-electron accumulation on one ormore pixels used to capture the first image of the scene, wherein thesecond exposure length is shorter than the first exposure length;capture, on a second subset of pixels on the image sensor, a secondimage of the scene with the second exposure length and third image ofthe scene with the second exposure length, wherein the second image andthe third image are captured sequentially during the capturing of thefirst image; and combine the first image, the second image, and thethird image to form a high dynamic range image of the scene.
 21. Thenon-transitory computer readable storage medium of claim 20, wherein thefirst exposure length is set at a length at which a certain proportionof pixels reach full-well capacity.
 22. The non-transitory computerreadable storage medium of claim 20, wherein the first subset of pixelscomprises a first plurality of pixel lines of a pixel array and whereinthe second subset of pixels comprises a second plurality of pixel linesof the pixel array.
 23. The non-transitory computer readable storagemedium of claim 22, wherein the first plurality of pixel lines and thesecond plurality of pixel lines are interlaced with each other.
 24. Thenon-transitory computer readable storage medium of claim 20, wherein thesecond image comprises a combined plurality of images with the secondexposure length.
 25. The non-transitory computer readable storage mediumof claim 20, wherein the first image and the second image each compriseat least one readout frame and one reset frame.
 26. The non-transitorycomputer readable storage medium of claim 20, wherein the rate ofphoto-electron accumulation is based on employing correlated doublesampling (CDS) to read out accumulate charge on the one or more pixelsused to capture the first image of the scene.